Since XXXX, Software and Data Carpentry have collected information on learner demographics, perception of tools and confidence in working with data. As we continue in our goal to streamline processes as The Carpentries, the Assessment Team completed an analysis of the pre- and post-workshop surveys for both Software and Data Carpentry. The goal of this analysis is to understand the impact our workshops are having on learners, and how we can improve our surveys and assessment infrastructure. This report covers the workshops from NA to NA.
As an overview, 1495 learners have responded to Data Carpentry’s surveys, while 14185 have responded to Software Carpentry’s.
This report includes the following:
Learners attend Carpentries workshops for many reasons. Data Carpentry’s workshops are domain specific and focus on the fundamental data skills needed to conduct research. Data Carpentry’s Ecology and Social Sciences curricula begin with a lesson on data organization and includes data cleaning with OpenRefine. From there, learners spend time learning a base programming language, either Python or R.
Data Carpentry’s Genomics curriculum also includes programming, however the focus of this curriculum is best practices for organization of bioinformatics projects and data, use of command line utilities and tools to analyze sequence quality and perform variant calling, and connecting to and using cloud computing.
| Why learners attend Data Carpentry workshops | n | % |
|---|---|---|
| To learn skills that I can apply to my work in the future | 960 | 63.7 |
| To learn skills that will help me get a job | 445 | 29.5 |
| As a requirement for my program/current position | 103 | 6.8 |
XX% of Data Carpentry learners attend workshops to learn skills they can apply to their work in the future.
Software Carpentry workshops teach automation with the Unix shell, a tool that allows users to run commands interactively or by scripting. In Software Carpentry workshops, version control of source code with Git and GitHub are also taught for learners to learn facilitating contribution and collaboration on online repositories. Programming in R or Python is also taught. Software Carpentry’s curriculum teaches basic lab skills for scientific computing. Compared to Data Carpentry’s learners, Software Carpentry’s tend to have more experience witht the tools covered in the workshops, and learners come to learn new and/or additional topics (XX%). It’s also interesting to note that XX% of Software Carpentry respondents are first-time learners.
| Why learners attend Software Carpentry Workshops | n | % |
|---|---|---|
| To cover new/additional topics | 458 | 74.7 |
| To network | 78 | 12.7 |
| To become a Software Carpentry helper/instructor | 39 | 6.4 |
| To help host/run a workshop | 38 | 6.2 |
| Software Carpentry 1st Time Learners | n | % |
|---|---|---|
| Yes | 11525 | 94 |
| No | 697 | 6 |
The majority (XX%) of Data Carpentry’s respondents are either unsatisfied or feel neutral about being satisfied with their current data management practices. By data management practices, we include behaviors such as keeping your raw data raw, reusing code, and using databases, queries, and scripts to manage large datasets.
| Data Carpentry Learners satisfaction with current data management practices | n | % |
|---|---|---|
| Very unsatisfied | 100 | 6.7 |
| Unsatisfied | 408 | 27.3 |
| Neutral | 405 | 27.1 |
| Satisfied | 173 | 11.6 |
| Very satisfied | 22 | 1.5 |
| Not sure | 75 | 5.0 |
| Not applicable | 60 | 4.0 |
| Didn’t answer | 252 | 16.9 |
In terms of current programming usage, XX% of learners either never use programming, or use programming less than once per year, but no more than several times per year. Only XX% program on a daily basis. This is no surprise, as Data Carpentry workshops tend to attract novices.
| Data Carpentry Learners current programming usage | n | % |
|---|---|---|
| Never | 314 | 31.9 |
| Less than once per year | 151 | 15.4 |
| Several times per year | 168 | 17.1 |
| Monthly | 101 | 10.3 |
| Weekly | 131 | 13.3 |
| Daily | 118 | 12.0 |
Data Carpentry workshops are mainly populated by “word-of-mouth”. XX% of respondents learned about the workshop from either a friend, colleague, or their advisor or supervisor.
| How respondents find out about Data Carpentry workshops | n | % |
|---|---|---|
| My friend/colleague told me about it | 324 | 47.0 |
| My advisor/supervisor told me about it | 244 | 35.4 |
| Read about it in a newsletter or university web site | 85 | 12.3 |
| Other web site | 20 | 2.9 |
| Twitter or other social media | 17 | 2.5 |
On the contrary, the majority of Software Carpentry workshop respondents (XX%) find out about workshops through instituion mailing lists or flyers.
| How respondents find out about Software Carpentry workshops | n | % |
|---|---|---|
| Institution mailing list or flyer | 5152 | 75.3 |
| Conference/meeting/seminar | 641 | 9.4 |
| Funding organization or program officer | 357 | 5.2 |
| Our website | 357 | 5.2 |
| Social Media (Twitter, Facebook, etc.) | 294 | 4.3 |
| Journal or publication | 37 | 0.5 |
In summary, both Data and Software Carpentry workshop respondents attend workshops to learn about or improve upon their current data management and analysis skills.
As previously mentioned, Data Carpentry workshops are domain specific, and curricula include Ecology, Genomics, Geospatial, Social Sciences, and Reproducible Research. XX% of respondents learned R in their workshop, while XX% learned Python.
| Data Carpentry: Language Covered in Workshops | n | % |
|---|---|---|
| R | 808 | 64.7 |
| Python | 188 | 15.1 |
| I don’t know/I don’t remember | 218 | 17.5 |
| Neither | 35 | 2.8 |
The Carpentries is committed to making participation in our workshops a harassment-free experience for everyone, regardless of who you are, where you come from, or your experience with the tools we teach. We establish norms for interaction by having, discussing, and enforcing a Code of Conduct such that our workshops provide open and inclusive learning environments. XX% of Data Carpentry respondents either agree or strongly agree that they felt comfortable learning in their workshop environment, and XX% of Software Carpentry’s respondents agreed or strongly agreed the workshop atmosphere was welcoming.
Data Carpentry respondents were asked to rate their level of agreement with several statements regarding their instructor’s knowledge, instructional method, and enthusiasm. Their responses are in the figure below, and axis labels correspond to the statements as follows:
The largest impact we see is that XX% of respondents said they felt comfortable interacting with the instructors. We know that our instructors are the reason why our workshops are so well received. It’s also great to see that XX% and XX% of respondents felt our instructors were knowledgeable about the material being taught, and were enthusiastic about the workshop, respectively. We would like to explore what training would help our instructors so that the percentage of respondents who felt they were able to get clear answers to their questions from the instructors would increase.
Software Carpentry respondents were asked to rate how they felt instructors and helpers worked as a team based on the following criteria:
The two Likert plots below provide an analysis of respondent’s answers for both instructors and helpers.
From the figures above, we see that Software Carpentry instructors and helpers are considerate, enthusiastic, give clear answers to questions, and are good communicators. As a whole, our instructors work as a team and are successful in creating a warm and welcoming workshop environment.
One of the goals for Data Carpentry’s lessons is that learners are able to immediately apply what they learned at the workshop. The figure below shows that XX% either agree or strongly agree that they’re able to immediately apply what they learned.
As the majority of Software Carpentry learners attend workshops to learn new skills it is great to see that XX% of learners either learned mostly or all new information during the workshop.
We want to be proactive in ensuring learners have access to whatever they need to participate in a workshop. Both Data and Software Carpentry learners are asked to inform workshop organizers if there is anything they need that would make their workshop experience better. Data Carpentry’s respondents were asked if they had accessibility issues, and XX% reported they did. After reading the open-ended responses, we see that the issues were related to not being able to hear and/or see in the back of the room.
| Data Carpentry Respondents Having Accessibility Issues | n | % |
|---|---|---|
| No | 654 | 89.2 |
| Yes | 79 | 10.8 |
We use the Net Promoter Score to measure learners’ likelihood of recommending workshops to a friend or colleague. The scoring for this question is on a 0 to 100 scale. Respondents scoring from 0 to 64 are labeled Detractors, and are believed to be less likely to recommend a workshop. Those who respond with a score of 85 to 100 are called Promoters, and are considered likely to recommend a workshop. Respondents between 65 and 84 are labeled Passives, and their behavior falls in the middle of Promoters and Detractors.
| Data Carpentry Promoter Score | n | % |
|---|---|---|
| Detractor | 32 | 4.395604 |
| Passive | 131 | 17.994505 |
| Promoter | 565 | 77.609890 |
78% of Data Carpentry respondants are promoters (i.e. would recommend a workshop).
| Software Carpentry Promoter Score | n | % |
|---|---|---|
| Detractor | 549 | 16.69708 |
| Passive | 634 | 19.28224 |
| Promoter | 2105 | 64.02068 |
For Software Carpentry respondents, 64% are promoters.
In summary, Data and Software Carpentry workshops provide a warm and welcoming environment whether learners are brand new to programming or have some experience. Attendees are recommending workshops to their friends and colleagues, and we know that our instructors and helpers are the major reason why.
In our workshops, we recommend that learners use their own machines. It is important for learners to leave the workshop with their own machine set up to do real work. Our instructors teach on three major platforms: UNIX/Linux, Mac OS X, and Windows. We see a very close representation of Windows (XX%) and Apple/Mac OS (XX%) users in our Data Carpentry workshops, and even a few UNIX/Linux users (XX%).
| OS Respondents Use in Data Carpentry Workshops | n | % |
|---|---|---|
| Windows | 661 | 53.3 |
| Apple/Mac OS | 512 | 41.3 |
| UNIX/Linux | 50 | 4.0 |
| Not sure | 17 | 1.4 |
Learners were asked to rate their level of agreement with the following statements related to Data Carpentry’s workshop goals and learning objectives. The figure below provides a visual representation of their responses, comparing them before the workshop and after the workshop. Axis labels and the corresponding question are as follows:
As the scoring for the above factors is ordinal from strongly disagree (1) to strongly agree (5), we show the mode (most frequent responses) for respondents’ before the workshop, and after the workshop. If there is only one point on the plot it is because the mode for pre- and post-workshop responses was the same. The comparison above is paired, meaning, we are comparing those who provided us with a unique identifier and who completed both the pre- and post-workshop survey. This figure includes XX respondses.
In the figures below we show another representation of the pre- and post-comparison of respondents skills and perspectives. The figures below include the data for all learners, not only those who provided a unique identifier and took both the pre- and post-workshop surveys. What we see is a shift in the distribution of for each factor, meaning, respondents’ self-reported confidence and ability shifted in a positive directions.
Another representation of the positive shift in distribution is provided below.
The neutral centered graphs below provide an even clearer picture of the shift in respondents self-reported confidence and skills.
It is interesting to see the shift in neutrality between the pre-workshop scores and post-workshops scores, especially for Programming Efficient. There was a higher percentage of learners beginning the workshop who felt programming with R or Python can make them more efficient at working with data. Contrarily, confidence in using programming to work with data increased from XX% to xx%.
Software Carpentry Respondets were asked to tell us about their experience with these topics before the workshop:
From the figure we see that XX% and less had extensive knowledge of the topics covered in their workshop.
The following is a comparison of Software Carpentry Respondents’ knowledge about the tools before compared to after the workshop. We see clearly that after the workshop, respondents’ knowledge of Git, Python, R, and the Unix Shell increased a great deal.
Motivation is important, but being confident in your ability to complete specific computing tasks is an equally important goal of Software Carpentry. The grid below shows respondents’ self-reported ability to complete tasks including:
It also provides their self-reported level of confidence in being able to complete the tasks above after completing the workshop.
These figures tell us that, before the workshop, between XX% and XX% of the respondents did not feel they could initialize a repository in Git, write a ‘for loop’ to automate tasks, use pipes to connect shell commands, write a SQL query, and/or write a unit test in R or Python. XX% of learners felt their confidence increased greatly with respect to importing a library or package in R or Python. We consider this significant as it is one of the fundamental skills that allows learners to be successful in the other areas mentioned above.
In summary, respondents experienced increased confidence in their ability to perform specific computing tasks and solve problems, or at least search for answers to problems, as a result of participating in Software and Data Carpentry workshops.
The Carpentries is a global community that has recognized the importance of bringing people to data through high-impact trainings. Thought the majority of Data Carpentry respondents attended a workshop in the United States of America (XX%), we see an increase in workshops in places like Ethiopia (XX%), Switzerland (XX%), and India (XX%).
In Software Carpentry’s pre-workshop survey, respondents are asked whether or not their workshop takes place in the United States. XX% of respondents attended a U.S. workshop.
| Software Carpentry Workshops in US | n | % |
|---|---|---|
| Yes | 6489 | 45.7 |
| No | 4650 | 32.8 |
| Didn’t answer | 3046 | 21.5 |
As previously mentioned, Data Carpentry’s curricula is domain specific to Ecology, Genomics, Geospatial, and the Social Sciences. We see this in the distribution of respondents by discipline. XX% are in the Life Science, while XX%, XX%, and XX% are in Agricultural or Environmental Sciences, Bioinformatics/Genomics, and Biomedical/Health Sciences, respectively.
| Data Carpentry’s Respondents by Discipline | n | % |
|---|---|---|
| Life Sciences | 444 | 22.0 |
| Agricultural or Environmental Sciences | 307 | 15.2 |
| Bioinformatics/Genomics | 292 | 14.5 |
| Biomedical/Health Sciences | 288 | 14.3 |
| Social Sciences | 122 | 6.0 |
| Mathematics or Statistics | 101 | 5.0 |
| Earth Sciences | 96 | 4.8 |
| Engineering | 91 | 4.5 |
| Computer Science | 88 | 4.4 |
| Business/Economics | 57 | 2.8 |
| Humanities | 53 | 2.6 |
| Physical Sciences | 53 | 2.6 |
| Library Sciences | 28 | 1.4 |
Software Carpentry’s respondent base also has a majority Life Sciences base (XX%), however we also see representation from those working in Psychology, High Performance Computing, and Chemistry.
| Software Carpentry’s Respondents by Discipline | n | % |
|---|---|---|
| Life Science - Organismal/systems (ecology, botany, zoology, microbiology, neuroscience) | 2698 | 21.0 |
| Life Sciences (Genetics, genomics, bioinformatics ) | 2683 | 20.9 |
| Mathematics/statistics | 945 | 7.4 |
| Physics | 803 | 6.2 |
| Planetary sciences (geology, climatology, oceanography, etc.) | 787 | 6.1 |
| Civil, mechanical, chemical, or nuclear engineering | 696 | 5.4 |
| Medicine and/or Pharmacy | 688 | 5.4 |
| Social sciences | 591 | 4.6 |
| Chemistry | 578 | 4.5 |
| Economics/business | 483 | 3.8 |
| Psychology | 418 | 3.3 |
| Library and information science | 375 | 2.9 |
| High performance computing | 365 | 2.8 |
| Humanities | 319 | 2.5 |
| Education | 264 | 2.1 |
| Space sciences | 163 | 1.3 |
As many of The Carpentries’ workshops are hosted on college campuses and other research-based communities, there is no surprise that the majority of respondents are Graduate Students (XX% - DC, XX% - SWC), Research Staff (XX% - DC, XX% - SWC), and Postdoctoral Researchers (XX% - DC, XX% - SWC).
| Data Carpentry’s Respondents by Position | n | % |
|---|---|---|
| Graduate Student | 592 | 45.9 |
| Research Staff | 200 | 15.5 |
| Postdoctoral Researcher | 183 | 14.2 |
| Faculty | 101 | 7.8 |
| Government Employee | 80 | 6.2 |
| Industry Employee | 49 | 3.8 |
| Undergraduate Student | 48 | 3.7 |
| Management/Administrator | 20 | 1.5 |
| Retired/Not Employed | 18 | 1.4 |
Gender and racial/ethnic identity information is collected for U.S. participants, as we are keen to increase the number of diverse instructors and learners we serve. Understanding our demographic makeup helps us to strategize about what communities to reach and what programs to offer.
Currently, both Data (XX%) and Software (XX%) Carpentry see strong representation from Women in the United States. Where we hope to improve is in reaching the non-White audience, as less than XX% of our respondents are from communities historically underrepresented in the science, technology, engineering, and mathematics (STEM) fields.
| Data Carpentry’s U.S. Respondents’ Gender Identity | n | % |
|---|---|---|
| Female | 322 | 58 |
| Male | 223 | 40 |
| Transgender female | 2 | 0 |
| Prefer not to answer | 8 | 1 |
| Data Carpentry’s U.S. Respondents Racial/Ethnic Identity | n | % |
|---|---|---|
| White | 316 | 54.0 |
| Asian | 152 | 26.0 |
| Hispanic or Latino(a) | 57 | 9.7 |
| I prefer not to say. | 28 | 4.8 |
| Black or African American | 25 | 4.3 |
| American Indian or Alaska Native | 4 | 0.7 |
| Native Hawaiian or Other Pacific Islander | 3 | 0.5 |
| Software Carpentry’s U.S. Respondents’ Gender Identity | n | % |
|---|---|---|
| Female | 322 | 58 |
| Male | 223 | 40 |
| Transgender female | 2 | 0 |
| Prefer not to answer | 8 | 1 |
| Software Carpentry’s U.S. Respondents’ Racial/Ethnic Identity | n | % |
|---|---|---|
| American Indian or Alaskan Native | 16 | 0 |
| Asian / Pacific Islander | 751 | 23 |
| Black or African American | 144 | 4 |
| Hispanic or Latino | 204 | 6 |
| Native Hawaiian or Other Pacific Islander | 3 | 0 |
| White / Caucasian | 1916 | 58 |
| Prefer not to say | 197 | 6 |
| Multiple ethnicity / Other (please specify) | 95 | 3 |